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Executive Summary 


Usage of Policies and Practices Promoted by RTT and SIG 


EXECUTIVE SUMMARY 

In response to the Great Recession, the U.S. Congress passed, and President Barack Obama 
signed into law, the American Recovery and Reinvestment Act of 2009 (Pub. Law 1 1 1-5). At an 
estimated cost of $831 billion, this economic stimulus package sought to save and create jobs, 
provide temporary relief to those adversely affected by the recession, and invest in education, 
health, infrastructure, and renewable energy. Education received a total of $100 billion, including 
$7 billion for two of the Obama administration’s signature grant programs: Race to the Top 
(RTT) and School Improvement Grants (SIG). RTT competitively awarded grants to states who 
agreed to implement a broad range of education policies and practices promoted by the program. 
SIG competitively awarded grants to the nation’s lowest-performing schools who agreed to 
implement a comprehensive set of improvement practices promoted by the program. While RTT 
focused on state policies and SIG focused on school practices, both programs promoted related 
policies and practices, and included a particular emphasis on turning around the nation’s lowest- 
performing schools. 

Given the sizable investment in both of these related programs, it is of policy interest to 
know (1) if states and schools that received grants are actually using the policies and practices 
promoted by these two programs; (2) if their usage of these policies and practices differ from 
states and schools that did not receive grants; and (3) if receipt of these grants ultimately 
impacted student achievement. Unfortunately, comprehensive evidence on these three 
implementation and impact questions has been limited to date. 

This report, part of the first large-scale evaluation of RTT and SIG, focuses on the first two 
questions and sets the table for a future report that will focus on the third question. RTT and SIG 
are complex programs that promote a broad range of policies and practices, some of which might 
be especially challenging to implement. Thus, this report’s findings on the first question should 
be useful to both policymakers considering the future direction of the programs and to program 
support staff, since it identifies areas that states and schools have and have not yet addressed in 
the two programs. This report’s findings on the second question can provide a useful policy 
context for impact findings in the future report. For example, if states and schools that received 
grants are similar to states and schools that did not receive grants in their usage of the policies 
and practices promoted by RTT and SIG, then it seems less likely that RTT and SIG would have 
a subsequent impact on student achievement. 

Race to the Top 

Four main findings, which are detailed in the first volume of this report, emerged for Race to 
the Top (all differences reported are statistically significant): 

• In five of the six major areas examined, early (Round 1 and 2) RTT states 
reported using more policies and practices promoted by RTT than non-RTT 
states in spring 2012. The five areas were (1) improving state capacity to support 
school improvement efforts; (2) adopting standards and assessments that prepare 
students to succeed in college and the workplace; (3) building state data systems that 
measure student growth and inform instruction; (4) recruiting, developing, rewarding, 
and retaining effective teachers and principals; and (5) encouraging conditions in 
which charter schools can succeed. The two groups of states did not differ in the 
extent to which they reported using policies and practices in the area of turning 
around low-perfonning schools. 
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• In one of the six major areas examined (teacher and principal certification and 
evaluation), later (Round 3) RTT states reported using more policies and 
practices promoted by RTT than non-RTT states in spring 2012. The two groups 
of states did not differ in the extent to which they reported using policies and 
practices in the other five areas examined. 

• Across all states, usage of policies and practices promoted by RTT was highest in 
the state capacity and data systems areas and lowest in the teacher and principal 
certification and evaluation area. In the state capacity and data systems areas, states 
reported using, on average, 66 and 68 percent of the RTT-promoted practices 
examined. In the teacher and principal certification and evaluation area, states 
reported using, on average, 24 percent of RTT-promoted practices examined. 

• There were no differences between RTT and non-RTT states in usage of English 
Language Learner (ELL)-focused policies and practices promoted by RTT. 

Within each group of states (early RTT, later RTT, non-RTT), states with higher 
percentages of ELLs used more ELL-focused policies and practices than states with 
lower percentages of ELLs, but there were no differences in usage between states 
with higher and lower ELL/non-ELL achievement gaps. 

School Improvement Grants 

Three main findings, which are detailed in the second volume of this report, emerged for 
School Improvement Grants (all differences reported are statistically significant): 

• In all four areas examined, schools implementing a SIG-funded model reported 
using more practices promoted by SIG in spring 2012 than schools not 
implementing such models. These results were found in four areas: (1) 
implementing comprehensive instructional reform strategies, (2) developing and 
increasing teacher and principal effectiveness, (3) increasing learning time and 
creating community-oriented schools, and (4) having operational flexibility and 
receiving support. 

• Across all schools (including those implementing a SIG-funded model and those 
not implementing one), usage of practices promoted by SIG was highest in the 
comprehensive instructional reform strategies area and lowest in the operational 
flexibility and support area. In the comprehensive instructional reform strategies 
area, study schools reported using, on average, 90 percent of the SIG-promoted 
practices examined. In the operational flexibility and support area, study schools 
reported using, on average, 46 percent of the SIG-promoted practices examined. 

• There were no differences between schools implementing a SIG-funded model 
and schools not implementing one in usage of ELL-focused practices promoted 
by SIG. Within each group of schools (those implementing a SIG-funded model and 
those not implementing one), schools with higher percentages of ELLs used more 
ELL-focused practices than schools with lower percentages of ELLs, but there were 
no differences in usage between schools with higher and lower ELL/non-ELL 
achievement gaps. 

Together, the findings suggest that it is possible RTT and SIG influenced the use of these 
promoted policies and practices in RTT states and in schools implementing SIG-funded models. 
It is also possible that RTT and SIG had no influence. Lor example, RTT grants were awarded 
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based partly on states’ prior progress and success in creating conditions conducive to education 
policy, and in fact some differences in usage of policies and practices promoted by RTT existed 
prior to states’ receipt of RTT grants. Similarly for SIG, most districts in the study reported 
taking schools’ prior academic achievement and ongoing improvement efforts into account in 
their applications. Some of the differences in usage of policies and practices promoted by SIG 
may have thus pre-dated SIG. Even though we are not able to determine the cause of differences 
in usage, it is still of value to policymakers to know where states and schools stand with their 
usage. These findings will also provide valuable policy context for interpreting the impact 
findings on student achievement that will appear in a future report. 

STUDY OF POLICIES AND PRACTICES USED BY RTT STATES AND NON-RTT 
STATES 

Background 

The first three rounds of the RTT grant competition sought to encourage states to 
implement a range of policies and practices designed to affect all levels of the education 
system, with the ultimate goal of improving student outcomes. 1 The six topic areas described in 
the RTT application criteria were (1) improving state capacity to support school improvement 
efforts; (2) adopting standards and assessments that prepare students to succeed in college 
and the workplace; (3) building state data systems that measure student growth and inform 
school staff about how they can improve instruction; (4) recruiting, developing, rewarding, 
and retaining effective teachers and principals; (5) turning around low-perfonning schools; 
and (6) encouraging conditions in which charter schools can succeed. The RTT objectives in 
each topic area and the subtopics within each topic for which we had state interview data 
are detailed in Exhibit ES.l. The RTT grants were awarded to states that both demonstrated a 
solid record of reform and presented strong plans in their RTT applications for furthering 
policies in these areas. The Race to the Top Executive Summary (available at 
http://www2.ed.gov/programs/racetothetop/executive-summary.pdf) summarizes the program’s 
priorities and theory of action. 


1 Additional rounds of the RTT program focused on improving early learning and development programs for 
young children and supporting district-developed plans to improve student achievement. Those rounds were not a 
focus of this study. 
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Exhibit ES.1 . RTT Objectives and Subtopics Addressed by State Interview Questions, by Topic Area 

Improving state capacity to support school improvement efforts 

Articulating the state’s education reform agenda and local education agencies’ participation in it a 
Building strong statewide capacity to implement, scale up, and sustain the proposed plans 
Demonstrating significant progress in raising achievement and closing gaps 15 ’ 0 

Adopting standards and assessments that prepare students to succeed in college and the workplace 

Developing and adopting common standards 0 

Developing and implementing common, high-quality assessments 0 

Supporting the transition to enhanced standards and high-quality assessments 

Building state data systems that measure student growth and inform instruction 

Fully implementing a statewide longitudinal data system 
Accessing state data and using it to inform key stakeholders 
Using data to improve instruction 

Recruiting, developing, rewarding, and retaining effective teachers and principals 

Providing high-quality pathways to certification for aspiring teachers and principals 0 
Improving teacher and principal effectiveness based on performance 0 
Ensuring equitable distribution of effective teachers and principals 0 
Improving the effectiveness of teacher and principal preparation programs 0 
Providing effective support to teachers and principals 15 0 

Turning around low-performing schools 

Authority to intervene in the lowest-achieving schools and local education agencies 0 
Turning around the lowest-achieving schools 

Encouraging conditions in which charter schools can succeed 
Eliminating restrictions on charter school creation and enrollment 0 
Refining authorization and monitoring processes 
Source: RTT application; interviews with state administrators in spring 2012. 

ELL = English language learner. 

a Regular school districts are the most common type of local education agency. 

b The number of questions included in the state interview was purposefully limited to reduce the time it took to 
complete the interview. We worked with the Institute of Education Sciences and the RTT Program Office to assess 
their priorities for the types of questions to include. The interview did not include any questions about this objective. 

0 The interview did not include any questions about this objective for ELLs. 

Across the first three rounds of competition, 46 states and the District of Columbia applied 
for RTT grants, and 19 applicants received grants. 2 The Round 1 awards were made in March 
2010, Round 2 awards in August 2010, and Round 3 awards in December 2011. The 12 states 
selected in the first two rounds received awards ranging from $75 million to $700 million. 3 In 
the third round, which was open only to the nine finalists from the second round, awards were 
made to 7 states. Since these awards were smaller (ranging from $17 million to $43 million), the 


2 Alaska, North Dakota, Texas, and Vermont did not apply for RTT grants in any round. 

3 States’ award amounts varied based on their share of the nation’s school-age population and the budget they 
proposed in their application for accomplishing the specific plans they proposed. 
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U.S. Department of Education (ED) only required these states to focus on a portion of the 
policies described in their Round 2 applications. 

Research Questions and Study Design 

The portion of this report focused on RTT was guided by the following research questions: 

• Are RTT states using the educational policies and practices promoted by RTT, and 
how does that compare to the usage of those policies and practices by non-RTT 
states? 

• Does usage of these policies and practices include a focus on ELLs, and does that 
focus on ELLs vary between RTT and non-RTT states? Does usage of these ELL- 
focused educational policies and practices vary by the percentage of students who are 
ELLs or the ELL/non-ELL student achievement gap? 

The general theory of action for RTT is that policy changes at the state level represent the 
first step in the process of changing the education system and that the ultimate goal is improving 
student outcomes. Changes will most likely occur at the state level before occurring at other 
levels, such as districts, schools, and classrooms. The state interviews, which took place roughly 
25 months after Round 1 grants were received, 20 months after Round 2 grants were received, 
and 4 months after Round 3 grants were received, were thus designed to capture information 
about this first step in the overall policy process. The RTT study design is summarized in the box 
below. 
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RTT Study Design 

Comparison design. We used a comparison design in which we compared RTT states to non-RTT states for 
volume 1 of this report, “Usage of Policies and Practices Promoted by Race to the Top.” 

Sample. The sample for the RTT evaluation included 49 states and the District of Columbia. 

Data on educational policies and practices. In spring 2012, to collect information on the extent to which 
states were using the educational policies and practices promoted by RTT, we conducted structured telephone 
interviews with representatives from the state education agencies in 49 states and the District of Columbia. The 
interviews covered six main topic areas: (1) state capacity to support education reforms, (2) state data systems, 
(3) standards and assessments, (4) teacher and principal certification and evaluation, (5) school turnaround, and 
(6) charter schools. Interviews were organized into modules (one per topic area). Typically, we interviewed 
different respondents for each module. Respondents were chosen based on which state administrators were 
most knowledgeable about each topic area. Each question received only one response per state. The response 
rate for the state interviews was 98 percent (Texas did not participate in the interviews). To facilitate the study’s 
comparisons between RTT states and states that did not receive RTT grants, question wording was the same 
for both groups of states. The interviews conducted in spring 2012 collected data not only on policies and 
practices in the current school year (201 1-2012) but also baseline data about particular policies and practices in 
the year before the announcement and implementation of RTT (2007-2008). 

Analysis. To examine how the usage of educational policies and practices promoted by RTT compares 
between RTT states and non-RTT states, we conducted two types of comparisons: (1) early RTT states (Round 
1 and 2) with non-RTT states and (2) later RTT states (Round 3) with non-RTT states. We distinguish between 
early and later RTT states to account for differences in the grants’ timing, funding levels, and scope between 
these groups of RTT states. To summarize the large amount of data collected, we identified state interview 
questions that aligned with the policies and practices that RTT sought to affect. We determined the number of 
these policies and practices that each state reported using and then calculated the average number of policies 
and practices for early RTT states, later RTT states, and non-RTT states. We then tested whether differences 
between each of the RTT groups and the non-RTT group in the average number of policies and practices 
reported were statistically significant. Because the goal of this analysis was to provide descriptive information 
about the actual levels of policies and practices used by RTT and non-RTT states in spring 2012, the results 
were reported as raw (that is, unadjusted) means; they were not regression-adjusted to account for any pre- 
existing differences between RTT and non-RTT states. 


RTT States and Non-RTT States Differed Somewhat at Baseline 

Interpreting the differences between RTT and non-RTT states in spring 2012 requires 
understanding the characteristics of these two groups of states at baseline (prior to the awarding 
of RTT grants). The RTT program sought to reward states that not only proposed strong refonn 
plans but that also had a solid record of reform, so it is not surprising that the groups of states 
differed in the following baseline characteristics: 

• Relative to non-RTT states, early RTT states had used more of some RTT- 
promoted policies and practices before the grants were awarded. Early RTT 
states received higher scores from grant application reviewers than non-RTT states 
that applied for RTT on the portions of the RTT application focused on preexisting 
state policy conditions (218 versus 173 points) and early RTT states reported using 
more of the policies and practices aligned with the RTT program at baseline (2007- 
2008) than non-RTT states in the teacher and principal certification and evaluation 
area (not shown). 4 


4 When developing the interview protocol, we included baseline questions for particular policies and practices 
based on ED guidance about those in which it had interest in assessing change. We have baseline data for three of 
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• Relative to non-RTT states, later RTT states had used more of some RTT- 
promoted policies and practices before the grants were awarded. Later RTT 
states received higher scores from grant application reviewers than non-RTT states 
that applied for RTT on the portions of the RTT application focused on preexisting 
state policy conditions (205 versus 173 points), and later RTT states reported using 
more policies and practices aligned with the RTT program at baseline (2007-2008) 
than non-RTT states in the areas of teacher and principal certification and evaluation, 
and school turnaround (not shown). 

In Five of the Six Areas Examined, Early RTT States Reported Using More Policies and 
Practices Promoted by RTT Than Non-RTT States in Spring 2012 

In spring 2012, early RTT states, on average, reported using statistically significantly more 
policies and practices than non-RTT states in the following five areas (Figure ES.l) 5 : 

• State capacity (7.5 of 10 policies and practices reported by early RTT states compared 
to 6.2 for non-RTT states) 

• Standards and assessments (5.9 of 10 policies and practices reported by early RTT 
states compared to 4.0 for non-RTT states) 

• Data systems (6.3 of 8 policies and practices reported by early RTT states compared 
to 5.1 for non-RTT states) 

• Teacher and principal certification and evaluation (13.6 of 39 policies and practices 
reported by early RTT states compared to 7.4 for non-RTT states) 

• Charter schools (2.4 of 4 policies and practices reported by early RTT states 
compared to 1.5 for non-RTT states) 


(continued) 

the six areas examined: (1) teacher and principal certification and evaluation, (2) school turnaround, and (3) charter 
schools. 

5 Chapter IV of the RTT volume of the report contains results for the individual policies and practices within 
each of these areas. 
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Figure ES.1. States’ Reported Usage of Policies and Practices Promoted by RTT, by Topic Area, 
Spring 2012 



(10 items) Assessments (8 items) Principal Certification Turnaround Schools 

(10 items) and Evaluation (10 items) (4 items) 

(39 items) 


Early RTT States (N=12) 


Later RTT States (N=7) 


Non-RTT States (N=31) 


Source: Interviews with state administrators in spring 2012. 

Note: Chapter IV of the RTT volume of the report contains results for the individual policies and practices 

within each topic area. The total number of policies and practices differs by topic area. This figure reads 
as follows (using the first bar on the left as an example): early RTT states reported using 75 percent of 
the policies and practices in the state capacity area, or 7.5 out of 10 policies and practices examined in 
that area. 

* Significantly different from non-RTT states at the 0.05 level, two-tailed test. 


The only area in which the differences between early states and non-RTT states were not 
statistically significant was school turnaround. The magnitude of these significant differences 
was about one or two practices for all but one area (teacher and principal certification and 
evaluation) where the difference was larger (about six practices). Therefore, although these 
differences were statistically significant, they may not all be substantively important. 

As mentioned above, we have evidence that some of these differences predated award of the 
RTT grants. In the teacher and principal certification and evaluation area, for example, early 
RTT states not only reported using more of the policies and practices promoted by RTT than 
non-RTT states in spring 2012, but a similar pattern emerged for the 2007-2008 school year, 
before the RTT grants were awarded. In fact, among the three areas on which we collected 
baseline data on a subset of the policies and practices (teacher and principal certification and 
evaluation, school turnaround, and charter schools), we found a statistically significant difference 
between early RTT and non-RTT states prior to the RTT program in 2007-2008 for one area 
(teacher and principal certification and evaluation). This preexisting difference suggests that 
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RTT grants were awarded to states that were already using some of the promoted policies and 
practices rather than RTT awards causing the usage of those policies and practices. For three 
areas (state capacity, standards and assessments, data systems), we do not have information on 
usage of policies and practices prior to the awarding of RTT grants. Therefore, we cannot 
detennine if the observed differences between early RTT states and non-RTT states in these 
areas predated receipt of the RTT grant. 

In One of the Six Areas Examined, Teacher and Principal Certification and Evaluation, 
Later RTT States Reported Using More Policies and Practices Promoted by RTT Than 
Non-RTT States in Spring 2012 

In spring 2012, later RTT states, on average, reported using statistically significantly more 
policies and practices than non-RTT states in one of six areas promoted by RTT (the area of 
teacher and principal certification and evaluation) (Figure ES.l). 6 Later RTT states reported 
using 11.2 of 39 policies and practices in this area, compared to 7.4 for non-RTT states. The 
magnitude of this difference was just under 4 practices. As with the early RTT states, baseline 
data suggest that the groups differed somewhat in the teacher and principal certification and 
evaluation area before RTT grants were awarded. In particular, we found a statistically 
significant difference between later RTT and non-RTT states in this area prior to the RTT 
program in 2007-2008. 

Across All States, Usage of Policies and Practices Promoted by RTT Was Highest in the 
State Capacity and Data Systems Areas and Lowest in the Teacher and Principal 
Certification and Evaluation Area 

The highest levels of usage of RTT-promoted policies and practices were reported in the 
areas of state capacity and data systems. In those two areas, states reported using, on average, 66 
and 68 percent of the RTT-promoted practices examined (not shown). The lowest levels of usage 
of RTT-promoted policies and practices were observed in the teacher and principal certification 
and evaluation area. In that area, states reported using, on average, 24 percent of the RTT- 
promoted practices examined (not shown). When focusing on usage of individual policies and 
practices, the highest levels of usage were found for (1) having a state longitudinal data system 
(SLDS), and (2) having a process to identify teacher shortage areas, with average values of 0.94 
(not shown). 7 Six of the seven individual policies and practices with the lowest levels of usage 
were concentrated in the area of teacher and principal certification and evaluation (for example: 
(1) requiring teacher evaluations to inform decisions about career advancement, and (2) 
conducting analyses of effectiveness based on student achievement growth to detennine whether 
there has been a shift in the distribution of effective principals), and the seventh was giving 


6 Because Round 3 grants were narrower in scope, not all later RTT states focused on every area targeted by 
RTT. In addition, less time had elapsed between receipt of RTT awards and our spring 2012 interviews for the later 
RTT states than for the early RTT states. Because of these reasons and the smaller sample size (7 later RTT states, 
as opposed to 12 early RTT states), statistically significant differences are less likely to be found between later RTT 
states and non-RTT states than between early RTT states and non-RTT states. 

7 For each policy or practice in the RTT application criteria for which we identified one or more relevant 
interview questions, we constructed a variable ranging from zero to one using those questions. To calculate the 
average value of a policy or practice, we averaged that variable across all states. For example, if half the states used 
a particular practice (meaning they had a value of 1) and the other half of states did not use that practice (meaning 
they had a value of 0), the average value for that practice would be 0.5. 
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priority to schools that address the needs of ELLs when considering applications for new charter 
schools; each had an average value of 0.02 (not shown). 

There Were No Differences Between RTT and Non-RTT States in Usage of ELL-Focused 
Policies and Practices Promoted by RTT 

Early RTT states, later RTT states, and non-RTT states reported using, on average, about 
half of the 12 ELL-focused policies and practices aligned with the RTT application criteria (6.3, 
6.1, 5.6) (not shown). Within each of these three groups, states with higher percentages of ELLs 
used more ELL-focused policies and practices than states with lower percentages of ELLs, but 
there were no differences in usage of ELL-focused policies and practices between states with 
higher and lower ELL/non-ELL achievement gaps (not shown). 

RTT Conclusions 

The average number of RTT-promoted policies and practices in use in spring 2012 was 
significantly higher in early RTT states than non-RTT states in all but one of the six areas 
examined (school turnaround) and was significantly higher in later RTT states than non-RTT 
states in just one area (teacher and principal certification and evaluation). These findings suggest 
that the RTT program may have influenced the usage of policies and practices promoted by the 
programs in RTT states. However, one cannot conclude that the RTT program caused these 
observed differences. Other factors could explain the differences. In particular, some differences 
in usage of policies and practices promoted by RTT existed prior to states’ receipt of RTT grants. 
In addition, later RTT states had their grants for a short period of time as of spring 2012 (Round 
3 grants were awarded in December 2011). Because these types of policies can take time to 
implement, a future report, based on spring 2013 data, will examine usage of policies and 
practices promoted by RTT one year later. 

STUDY OF PRACTICES USED IN SCHOOLS IMPLEMENTING A SIG-FUNDED 
INTERVENTION MODEL COMPARED WITH SCHOOLS NOT IMPLEMENTING SUCH 
A MODEL 

Background 

The SIG program aims to support the implementation of school intervention models in low- 
performing schools. Although SIG was first authorized in 2001, this evaluation focuses on SIG 
awards granted in 2010, when roughly $3.5 billion in SIG awards were made to 50 states and the 
District of Columbia, $3 billion of which came from the American Recovery and Reinvestment 
Act of 2009. The four turnaround models and requirements for those models were the same 
under Race to the Top and School Improvement Grants. States identified the low-performing 
schools eligible for SIG based on criteria specified by ED and then held competitions for local 
education agencies seeking funding to help turn around eligible schools. 

For the 2010 SIG competition, ED required states to categorize eligible schools into three 
eligibility tiers based on the school’s level (elementary or secondary), Title I status, 8 and 


8 Title I, Part A (Title I) of the Elementary and Secondary Education Act provides financial assistance to local 
educational agencies and schools with high numbers or high percentages of children from low-income families. A 
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achievement or graduation rate. These tiers helped prioritize the distribution of SIG at the local 
level and detennined the practices to be used for school turnaround. In general, SIG eligibility 
Tiers I and II included schools with the lowest and most persistent achievement problems in each 
state. 

ED required that each SIG-awarded school under Tier I or Tier II implement one of four 
school intervention models. These models featured specific practices: 

• Transformation. This model requires schools to replace the principal, adopt a 
teacher and principal evaluation system that accounts for student achievement growth 
as a significant factor, adopt a new governance structure, institute comprehensive 
instructional reforms, increase learning time, create community-oriented schools, and 
have operational flexibility. 

• Turnaround. This model requires schools to replace the principal, replace at least 50 
percent of the school staff, institute comprehensive instructional reforms, increase 
learning time, create community-oriented schools, and have operational flexibility. 

• Restart. This model requires schools to convert to a charter school or close and 
reopen under the management of a charter management organization or education 
management organization. 

• School closure. This model requires districts to close schools and enroll their 
students in higher-achieving schools within the district. 

The practices laid out for the transformation and turnaround intervention models can be 
grouped into four broad areas set forth in the SIG application criteria: (1) implementing 
comprehensive instructional refonn strategies, (2) developing and increasing teacher and 
principal effectiveness, (3) increasing learning time and creating community-oriented schools, 
and (4) having operational flexibility and receiving support. 9 The SIG objectives in each topic 
area and the subtopics within each topic for which we had school administrator survey questions 
are detailed in Exhibit ES.2. 

Research Questions and Study Design 

The portion of this report focused on SIG was guided by the following research questions: 

• Are SIG-funded intervention model schools using the improvement practices 
promoted by the four SIG intervention models, and how does that compare to the 
usage of those practices by schools not implementing a SIG-funded intervention 
model? 

• Does usage of these practices include a focus on ELLs and does that focus on ELLs 
vary between schools implementing a SIG-funded intervention model and schools not 
implementing one? Does usage of these ELL-focused improvement practices vary by 


(continued) 

school receiving Title I funds that fails to meet adequate yearly progress targets can be assigned a Title I status of 
“in need of improvement, corrective action, or restructuring.” 

9 Detailed information about reform practices for the other two intervention models (restart and closure) was 
not provided in the SIG application. 
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the percentage of students who are ELLs or the ELL/non-ELL student achievement 
gap? 

Exhibit ES.2. SIG Objectives and Subtopics Addressed by School Administrator Survey Questions, by Topic 
Area 

Implementing comprehensive instructional reform strategies 

Using data to identify and implement an instructional program 13 
Promoting the continuous use of student data 

Conducting periodic reviews to ensure that the curriculum is being implemented with fidelity 315 
Implementing a new school model (such as a themed academy) 313 

Providing supports and professional development to staff to assist ELLs and students with disabilities 
Using and integrating technology-based supports 13 
Tailoring strategies for secondary schools 13 

Developing and increasing teacher and principal effectiveness 

Using rigorous, transparent, and equitable evaluation systems 13 

Identifying and rewarding effective teachers and principals and removing ineffective ones 13 
Providing high-quality, job-embedded professional development or supports 13 
Implementing strategies to recruit, place, and retain staff 

Increasing learning time and creating community-oriented schools 

Increasing learning time 13 

Engaging families and communities and providing a safe school environment that meets students’ social, 
emotional, and health needs 

Having operational flexibility and receiving support 

Having operational flexibility 13 
Receiving technical assistance and support 
Source: SIG application; surveys of school administrators in spring 2012. 

ELL = English language learner. 

3 The number of questions included in the school administrator survey was purposefully limited to reduce the time it 
took to complete the survey. We worked with the Institute of Education Sciences and the SIG Program Office to 
assess their priorities for the types of questions to include. The survey did not include any questions about this 
objective. 

b The survey did not include any questions about this objective for ELLs. 
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The SIG study design is summarized in the box below. 


SIG Study Design 

Comparison design. For the analyses conducted in volume two of this report, “Usage of Practices Promoted by 
School Improvement Grants,” we used a comparison design in which we compared schools implementing a SIG- 
funded model to similar schools in the same districts not implementing such a model. 

Sample. The sample for the SIG evaluation includes approximately 470 schools located in approximately 60 
districts and 22 states. The SIG sample was not randomly selected, rather schools were purposively selected to 
support the estimation of impacts of SIG-funded models on student outcomes which will be presented in a future 
report. Thirty-five percent of all schools implementing a SIG-funded model in 2011-2012 are in the sample. 
Because the SIG sample was not randomly selected, it should not be assumed that the SIG implementation 
findings presented in this report necessarily generalize to schools nationwide. 

Data on school improvement practices. In spring 2012, to collect information on the extent to which schools 
were using the improvement practices promoted by SIG, we conducted structured web surveys of roughly 290 
administrators of schools implementing a SIG-funded model and 180 administrators of similar schools in the 
same districts that were not implementing a SIG-funded model. The survey was sent to school principals, with 
instructions indicating that they should consult with other school staff if needed. The response rate for the school 
surveys was 87 percent. 

Analysis. To summarize the large amount of school survey data collected, we identified questions that aligned 
with the practices SIG sought to affect. We determined the number of these practices that each school reported 
using and then calculated the average number of practices for both sets of schools — those implementing SIG- 
funded models and those not doing so. We then tested whether differences between the two groups in the 
average number of practices reported were statistically significant. Because the goal of this analysis was to 
provide descriptive information about the actual levels of practices used by schools implementing a SIG-funded 
model in 2011-2012 and schools not implementing one, the results were reported as raw (that is, unadjusted) 
means; they were not regression-adjusted to account for any pre-existing differences between these two groups 
of schools. 


At Baseline, the Study’s SIG-Funded Model Schools Were Generally Similar to Study 
Schools That Were Not Implementing Such Models, But They Were Not Representative of 
All U.S. Schools Implementing These Models 

Interpreting the differences between schools implementing SIG-funded models and schools 
not implementing these models in spring 2012 requires understanding the characteristics of these 
two groups of schools at baseline (during the 2009-2010 school year, which was prior to SIG 
funding receipt). 

• The two groups of study schools generally had similar characteristics prior to 
the receipt of the 2010 SIG awards. 10 Out of five characteristics examined — 
race/ethnicity, percentage of students eligible for free or reduced-price lunch, Title I 
eligibility, location, and school level (elementary, middle, high school) — schools 
implementing a SIG-funded model and schools not implementing such a model 
differed by a statistically significant margin on one characteristic. Compared to 
schools not implementing a SIG-funded model, schools that were implementing such 
models had a statistically significantly higher percentage of students eligible for free 
or reduced-price lunch at baseline (84 percent versus 80 percent) (not shown). 


10 The baseline characteristics examined came from the Common Core of Data. To limit respondent burden, the 
school administrator survey focused primarily on practices being used in the schools in spring 2012. 
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• Study schools implementing a SIG-funded model were generally not 
representative of all U.S. schools implementing such models. The schools in our 
study that were implementing SIG-funded models were more disadvantaged and more 
likely to be in an urban area than U.S. schools nationally that were implementing such 
models. In particular, study schools implementing a SIG-funded model were more 
likely than U.S. schools nationally implementing such models to be eligible for Title I 
(95 percent versus 89 percent) and were more likely to be located in an urban area (87 
percent versus 59 percent) (not shown). Because the SIG sample is not representative 
of schools nationwide, readers should use caution when interpreting the results. 

In All Four Areas Examined, Schools Implementing a SIG-Funded Model Reported Using 
More Practices Promoted by SIG in Spring 2012 Than Schools Not Implementing Such 
Models 

In spring 2012, schools implementing a SIG-funded model reported using more practices 
promoted by SIG, on average, than schools not implementing such models, in the following 
areas (Figure ES.2) 11 : 

• Implementing comprehensive instructional refonn strategies (7.3 of 8 practices 
reported by SIG-funded model schools compared to 7.0 for schools not implementing 
such models) 

• Developing and increasing teacher and principal effectiveness (11.1 of 20 practices 
reported by SIG-funded model schools compared to 9.6 for schools not implementing 
such models) 

• Increasing learning time and creating community-oriented schools (3.9 of 5 practices 
reported by SIG-funded model schools compared to 3.2 for schools not implementing 
such models) 

• Having operational flexibility and receiving support (1.0 of 2 practices reported by 
SIG-funded model schools compared to 0.8 for schools not implementing such 
models) 

The magnitude of these differences was less than one practice for three areas and less than 
two practices for one area. Therefore, although these differences are statistically significant, they 
may not be substantively important. As noted above, because the SIG sample was not randomly 
selected, it should not be assumed that these findings necessarily generalize to schools 
nationwide. 

Usage of Practices Promoted by SIG Was Highest in the Comprehensive Instructional 
Reform Strategies Area and Lowest in the Operational Flexibility and Support Area 

The highest level of usage of SIG-promoted practices was observed in the area of 
comprehensive instructional refonn strategies. In that area, study schools reported using, on 
average, 90 percent of the SIG-promoted practices examined (not shown). The lowest level of 
usage of SIG-promoted practices was observed in the area of operational flexibility and support. 


11 Chapter IV of the SIG volume of the report contains results for the individual practices within each of these 

areas. 
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Study schools reported using, on average, 46 percent of the SIG-promoted practices examined in 
that area (not shown). When focusing on usage of individual practices, the highest level of usage 
was for using benchmark or interim assessments at least once per year, with an average value of 
0.99 (not shown). 12 The individual practice with the lowest level of usage was using teacher 
evaluation results to infonn decisions about compensation, with an average value of 0.15 (not 
shown). 

Figure ES.2. Study Schools’ Usage of Practices Promoted by SIG, by Topic Area, Spring 2012 



Comprehensive 
Instructional Reform 
Strategies 
(8 items) 


Teacher and 
Principal 
Effectiveness 
(20 items) 


Learning Time and 
Community-Oriented 
Schools 
(5 items) 


Operational 
Flexibility 
and Support 
(2 items) 


■ Schools Implementing a SIG-Funded Intervention Model in 2011-2012 (N=290) 
Schools Not Implementing a SIG-Funded Intervention Model in 2011-2012 (N=180) 


Source: Surveys of school administrators in spring 2012. 

Note: Chapter IV of the SIG volume of the report contains results for the individual practices within each topic 

area. The total number of practices differs by topic area. This figure reads as follows (using the first bar 
on the left as an example): schools implementing a SIG-funded intervention model reported using 91 
percent of the practices in the comprehensive instructional reform strategies area, or 7.3 out of 8 
practices examined in that area. 

* Significantly different from schools not implementing a SIG-funded intervention model in 2011-2012 at the 0.05 
level, two-tailed test. 


12 For each practice in the SIG application criteria for which we identified one or more relevant survey 
questions, we constructed a variable ranging from zero to one using those questions. To calculate the average value 
of a practice, we averaged that variable across all study schools. For example, if half the schools used a particular 
practice (meaning they had a value of 1 ) and the other half of schools did not use that practice (meaning they had a 
value of 0), the average value for that practice would be 0.5. 
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There Were No Differences Between Schools Implementing a SIG-Funded Model and 
Schools Not Implementing One in Usage of ELL-Focused Practices Promoted by SIG 

Schools implementing a SIG-funded model and schools not implementing one both reported 
using an average of 2.9 of the 6 ELL-focused practices aligned with the SIG application criteria 
(not shown). Within each of these two groups, schools with higher percentages of ELLs used 
more ELL-focused practices than schools with lower percentages of ELLs, but there were no 
differences between schools with higher and lower ELL/non-ELL achievement gaps (not 
shown). 

SIG Conclusions 

In all areas examined, usage of practices promoted by SIG in spring 2012 was higher in 
schools implementing a SIG-funded model than in schools not implementing such a model. 
These findings suggest that the SIG program may have influenced the usage of practices 
promoted by the program in schools implementing a SIG-funded model. However, one cannot 
conclude that the SIG program caused the observed differences in practices. As noted above, the 
findings should be interpreted with caution because the SIG sample was not randomly selected 
and is not representative of schools nationwide. Because the types of school improvement 
practices promoted by SIG can take time to implement, a future report based on spring 2013 data 
will examine usage of SIG-promoted practices one year later. 


16 





